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Introduction 

D uval County Florida superintendent John 
Fryer was not used to flying blind. A 
retired Air Foree major general and 
former taetieal fighter wing commander new to 
school district leadership, Fryer was used to 
having a control panel packed with information. 
In front of him in the cockpit, Fryer had a wide 
range of continually updated data from which he 
could adjust his flight path and inform his 
actions. “When you’re a fighter pilot, you rely 
on that information to guide your decision 
making,” he said. Not having any equivalent 
instruments in his position as superintendent. 
Fryer wondered, “Flow do I know whaf s 
happening in our school district?” (personal 
interview, January 1 1, 2000). In his effort to 
remedy this problem. Fryer sought a dashboard 
of data indicators to keep him updated on his 
district’s reform efforts. 

Fryer was hired in 1998 as the superintendent of 
Duval County, one of the 20 largest school 
districts in the United States. Fryer’s tenure is 
notable for his tenacious efforts to implement a 
particular vision of standards-based instructional 
practice across the district. To spearhead the 
district’s efforts. Fryer forged a distinctive 
partnership with the National Center on 
Education and the Economy (NCEE), which 
provides tools and expertise to build teachers’ 
and school leaders’ capacity to deliver 
standards-based instruction within a framework 
of comprehensive school reform. Concurrent 
revisions of the district’s mathematics and 
science curricula and professional development 
systems, with the assistance of strong local 
leadership and support from the National 
Science Foundation, helped to form a coherent 
and philosophically compatible change program. 
These efforts have resulted in significant and 
sustained improvements in the district’s test 
performance relative to comparable counties in 
Florida (Supovitz & Snyder, 2003). 

Throughout their journey. Fryer and his 
leadership team have persistently searched for 
ways to build a dashboard of indicators that 
provide data on the influence of the district’s 



strategic changes on the practices of the 
district’s 7,300 teachers, the test performance of 
the district’s 127,000 students, and the 
leadership of the district’s 149 schools. The 
superintendent developed a room in the school 
board building, dubbed the “mission control 
center,” in which indicators of district progress 
on its five strategic goals (academic 
performance, safe schools, accountability, 
learning communities, and high performance 
management teams) are tracked. The vision of 
the dashboard also led Duval County education 
leaders to develop their own system to monitor 
schools’ implementation of the district’s 
standards-based reform efforts. The system is 
called the Standards Implementation Snapshot 
System. The snapshot system seeks to take a 
“snapshot” at a point in time of the depth of 
implementation of the district’s standards-based 
reform initiatives. The system was conceived in 
the summer of 2002, and at the time of this 
report is completing its second year of use 
across the district. This is the story of the 
development and influence of the snapshot 
system. 

Fryer’s search for dashboard indicators is 
indicative of a more generalized set of problems 
with which education leaders have long 
struggled. Foremost, education leaders lack a 
clear, detailed, and timely perspective on what is 
happening in schools and classrooms as a 
consequence of their reform initiatives 
(Eeithwood & Aitken, 1995). If results are weak 
or mixed (as is often the case), leaders are 
typically unable to distinguish between 
ineffective reform ideas and poor 
implementation exactly because they lack an 
accurate picture of the depth of implementation. 
Argyris and Schon (1974; 1978) embodied this 
idea in their distinction between theory of 
action, or the logic advanced by advocates of 
reform to explain how an initiative is supposed 
to bring about intended results, and theory in 
use, how programs or policies are actually 
carried out. In essence, in order for reformers to 
advance understanding of the efficacy of their 
reform ideas, they map their theory in use 
against their theory of action. This mapping 
process reveals to what extent problems are due 
to a problematic theory of action, or whether 
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weak implementation, the theory in use, is the 
source of poor, or mixed, results. 

From a different perspective and tradition, the 
strategic planning literature has long 
incorporated a similar call for evaluation of 
activities and results relative to goals. For 
example, Koteen (1989) laid out the sequence of 
strategic planning as mission formulation, 
strategy determination, action specification, and 
appraisal of results. Similar approaches, such as 
Management by Objectives (MBO) and other 
goal-setting plans contain comparable 
monitoring components (Caroll, 2000). More 
recent formulations like Balanced Scorecard 
include a similar emphasis on the monitoring of 
implementation and impact (Kaplan & Norton, 
1996). The incorporation of these organizational 
monitoring and accountability strategies, 
commonly used in the business and nonprofit 
worlds, are also pervasive on the business 
management side of educational organizations. 
Facilities management, budgeting, and inventory 
monitoring are some of the areas to which the 
monitoring component of strategic planning is 
frequently applied in education. 

Some components of the strategic planning 
cycle are also applied to instructional delivery. 

Y et systematic implementation monitoring of 
instructional improvement efforts is often 
missing or weak. Districts and schools 
commonly formulate missions that incorporate 
instructional goals, and determine strategies to 
achieve their missions. They then monitor 
student performance results to determine the 
effectiveness of their strategies. This sequence is 
quite explicit in the school improvement 
planning process that is commonplace in schools 
across the country, although the instructional 
goals in these missions often lack the specificity 
required for adequate monitoring and assessment 
of results (Supovitz & Klein, 2002). The 
proliferation of student outcome data and 
systems to manipulate it, spurred by the No 
Child Left Behind Act of 200 1 , is a testament to 
how commonplace this means of assessing the 
achievement of instructional goals has become. 

Y et one of the most essential phases of strategic 
planning — monitoring of the implementation of 
instructional delivery — has rarely been 



systematically applied to monitoring of the 
attainment of instructional goals. 

Why haven’t instructional delivery systems 
undergone the same systematic scrutiny as have 
the managerial functions within education 
organizations? This may be attributable to the 
belief that instruction is not considered a 
routine-based activity, and therefore is not 
perceived to be readily monitored against a set 
of standard indicators (Elmore, 1993; Floden et 
al., 1988; Meyer & Rowan, 1978). An 
alternative hypothesis is the historical and 
cultural autonomy that teachers have typically 
held over instructional delivery (Lortie, 1975; 
Weick, 1976). Weick coined the term ’’loose 
coupling” to describe this phenomenon and the 
resulting uncertain relationship between the 
organization of education and its outcomes. In a 
highly autonomous system where teachers are 
seen to know best what their students need and 
are best left to determine how to teach them, 
there is no perceived need to monitor 
implementation, only results. 

Others view the implementation of complex 
technologies such as instructional reforms as a 
problem of organizational learning. In this view, 
the monitoring of the implementation of an 
innovation becomes a measure of the spread of 
knowledge throughout the organization. 
Organization theorists have likened the 
challenge of strategy implementation to one of 
creating the organizational learning required to 
implement the strategy (Gillen, 2000). In one 
view, implementation essentially becomes a 
problem of teaching people with sufficient 
specificity the knowledge required to implement 
a reform. This is both because a larger group of 
people are required to implement the strategy 
(Gillen, 2000) and because those on the ground 
who are critical to the policy’s success have had 
little, if anything, to do with its formulation 
(Wheelen & Hunger, 1997). An alternative 
strain of organizational learning views 
embedding a culture of systematic inquiry and 
continually deeper learning about what it takes 
to enact reform as the real engine of meaningful 
change (Fullan, Bertani, & Quinn, 2004; 
Supovitz & Klein, 2002). 
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The complexity of teaching and learning, the 
history of teacher autonomy, the loosely 
coupled organization of schools and school 
districts, and the tremendous challenge of deeply 
entrenching organizational learning into an 
organizational point to the difficulties of both 
implementing large-scale instructional reforms, 
and developing systems to monitor them. To do 
so implies, as is the case in Duval County, that 
district leaders have a distinct and fairly well 
specified vision of instructional reform that they 
seek teachers across the system to employ. The 
administrative sanction of a particular 
curriculum and form of instructional delivery 
legitimates the need to systematically monitor its 
implementation. 

In Duval County, district leaders advocate a 
particular brand of standards-based instructional 
delivery. All teachers are supposed to be 
knowledgeable of the expectations for student 
performance expressed in the Florida Sunshine 
State Standards and the New Standards 
Performance Standards. Literacy teachers are 
expected to use readers and writers workshop 
structures to engage students in authentic 
reading and writing experiences in a variety of 
genres. District leaders advocate inquiry-based 
activities in both mathematics and science. 
Mathematics curriculum is organized around the 
Investigations curriculum in the elementary 
grades, the Connected Mathematics curriculum 
in the middle grades, and the College 
Preparatory Mathematics curriculum in high 
school. The science curriculum is based upon a 
comprehensive approach structured around 
materials approved by the National Science 
Foundation. The district advocates an array of 
particular strategies to support students needing 
additional assistance. Teachers and school 
leaders are expected to use student performance 
data to inform their decision making. These 
district-wide expectations form a common basis 
against which implementation can be measured. 

In this article, we describe Duval’s system to 
monitor the district’s instructional reform efforts 
and the influences of the system on teachers and 
school and district leaders. First, we describe 
how the snapshot system is designed to work 
and articulate the distinguishing characteristics 



of the snapshot system. Then, we briefly discuss 
the design, data sources, and analytic methods 
used for this study. We then describe the effects 
of the snapshot system on district efforts, as 
reported by principals and district 
administrators. We conclude the article by 
discussing the implications of such an 
implementation monitoring system for district 
reform efforts. 

How the Snapshot 
System Works 

Snapshots are pictures at a particular point in 
time of the depth to which schools in Duval 
County are implementing key elements of the 
district’s reform vision. The snapshot system 
represents a true co-development project 
between leaders of the Duval County Public 
Schools and researchers at the Consortium for 
Policy Research in Education (CPRE)‘. In this 
uniquely collaborative effort, members of both 
organizations have taken ownership of the 
process, contributing their expertise and 
influence. In this section, we briefly summarize 
how the snapshot system is designed to work. 

In the summer of each year, the district’s 
superintendent and leadership team develop a 
list of three to five topics that are candidates for 
snapshots for the upcoming year. The topics are 
carefully chosen to reflect key elements of the 
district’s Framework for the Implementation of 
Standards, which articulates the district’s vision 
for standards-based reform, and the district’s 
priorities for the upcoming year. Topics are also 
sequenced to follow the district leadership 
training schedule. 

Each of the selected topics becomes the focus of 
two sets of school visits during the upcoming 
year. However, well before school visits occur, a 
team of district “experts” on a particular topic 
are brought together to develop a rubric on that 
area for the snapshot school visit. Snapshot 
topics in Duval County to date have included 
Understanding and Using Standards, Connecting 
Student Work to Standards in Reading, 
Connecting Student Work to Standards in 
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Mathematics, Safety Nets, Data-Driven Decision 
Making, and Professional Learning 
Communities. The rubric development team 
specifies the characteristics at the school, 
classroom, and student levels as to what 
comprises different levels of implementation. 

The rubric development team also constructs an 
evidence form, which is a short list of questions 
for school leaders, teachers, and/or students, as 
well as a list of artifacts for which the data 
collectors should look. These questions and 
artifacts are carefully designed to give 
opportunities for respondents to provide 
evidence of implementation, as represented on 
the rubric. The rubrics are then circulated 
throughout the district and vetted. A sample 
rubric and evidence form are provided in 
Appendix A. 

Finally, before the actual snapshot, the 
principals and district administrators who make 
up the snapshot data collection teams attend a 
half-day training in which they “practice” the 
snapshot in a small number of schools trying out 
the rubrics and evidence forms. After the 
training, they return to a central location for a 
debriefing and clarification session. Based on 
this and other collaborative feedback, 
modifications to the rubric and evidence form s 
are made. 

Snapshots occur monthly throughout the school 
year and are conducted by two to three trained 
principals and district administrators (depending 
on school size). Each month, this cadre of 
trained principals and district administrators, 
called snapshot data collectors, visit a 
representative sample of schools in the district to 
collect data on a particular element of the 
district’s reform efforts. The data collectors are 
selected by district administrators based on their 
knowledge of standards as well as their 
representation of the district as a whole. 

The sample of schools they visit is carefully 
chosen to represent the district in terms of prior 
achievement, reform experience, grade ranges, 
and region within the district. The sample can, 
therefore, be regarded as representative of the 
district as a whole. In 2002-2003, the first year 
of the snapshot system, all schools were visited 



once during the school year. In 2003-2004, the 
samples were increased so that each school was 
visited twice during the school year. 

The snapshot data collection teams visit each 
school in the sample for approximately three 
hours. As previously described, the visit is 
focused upon a particular topic that is central to 
the district’s reform efforts. In addition to the 
rubric specifying different stages of 
implementation of that particular element of the 
district’s reform efforts, each snapshot team is 
equipped with instructions on how to carryout 
their visit, a list of people to talk to (who could 
be principals, leadership team members, 
teachers, and/or students, depending on the 
focus of the snapshot), a prespecified set of 
questions to ask, and a defined set of artifacts to 
examine (see Appendix B). 

Meeting first with the school’s principal, the 
team develops a sampling frame of individuals 
and classrooms to visit and talk to in particular 
grades and subjects (depending somewhat on the 
topic of the snapshot). From this sampling 
frame, together with the principal they select a 
sample of teachers that is representative of the 
school. Team members may wish to split up the 
sample and conduct visits/interviews 
individually or conduct their visits/interviews as 
a team. They then spread out and collect the data 
upon which the snapshot is based. 

After completing the data collection, the team 
assesses the school on the areas outlined in the 
rubric, using the evidence collected during their 
visits/interviews. Using this evidence, the team 
makes judgments about the degree to which the 
school has implemented the components of the 
snapshot rubric. After coming to consensus, the 
team meets with the school’s principal to debrief 
and provide constructive feedback, sticking 
closely to what they observed and why they 
reached the conclusions that they did. In an 
effort to minimize inappropriate use, principals 
of host schools are provided with guidelines of 
appropriate ways they can use the feedback with 
their faculty. For the snapshot data collectors, 
the substance and conclusion of their visit is 
confidential and should not be discussed after 
they leave the host school. 
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A snapshot itself is completed within a window 
of approximately two weeks, which allows the 
snapshot data collectors and host school to 
schedule a visit time which is convenient for all. 
After completing their snapshot visit, a member 
of each snapshot team is asked to enter their 
ratings and comments onto a password-protected 
website. The results of the snapshot from the 
sample set of schools are aggregated to produce 
a picture of implementation of that particular 
topic across the district. Importantly, the 
aggregation provides anonymity to teachers and 
schools, reinforcing the stated purpose of the 
snapshots to capture district-wide depth of 
implementation of elements of the district’s 
frameworks. Graphical representations of the 
results and comments provided by the snapshot 
teams are produced. An excerpt of snapshot 
results is shown in Appendix C. 

The snapshot results are produced in time for 
monthly principals’ meetings. Before each 
principals’ meeting, the district’s five regional 
directors examine the results, discuss their 
meaning, and develop a set of guiding questions 
for principals to explore as they examine the 
snapshot results. During regional breakout 
sessions at the monthly principals’ meeting, the 
regional directors facilitate a conversation with 
their principals (there are approximately 35 
schools in each region), seeking to identify areas 
where the district is strongly implementing, 
barriers to implementation, and areas where 
implementation could be deepened, as well as 
seeking to cross-germinate and capture 
innovative strategies that schools are using. The 
regional directors take notes on their group’s 
conversation. These notes are compiled across 
the five regions and fed back to all principals 
and district training developers. 

Distinguishing Characteristics 
of the Snapshot System 

The snapshot system contains a set of key 
characteristics that distinguish it from other 
implementation monitoring systems, educational 
data systems, and strategic planning initiatives. 
In this section, we articulate some of the key 



characteristics of the snapshots and discuss how 

these differ from other systems. 

• Snapshots provide a picture of system-wide 
implementation of district reform efforts. 
The snapshot system is designed to capture 
the depth of implementation of reform 
efforts across the district at a particular 
point in time. No individual teachers or 
schools are identified when results are 
produced. The only picture provided is 
implementation across grade levels 
(elementary /middle/high) so that patterns of 
district-wide implementation are the focus. 
In this sense, this is a district accountability 
mechanism. 

• Snapshot topics are carefully aligned with 
district reform strategies. The topics for 
snapshots are carefully chosen so that they 
reflect district leaders’ priorities. In this way 
they signal to school leaders and teachers 
what district leaders care about. 

• Snapshot results are reliable and valid. 
Careful attention is paid to producing high 
levels of reliability and validity of the 
results. The sample of schools for each 
snapshot is deliberately chosen to reflect a 
representative sample of the district in terms 
of prior achievement (previous year’s state 
accountability grade), reform experience 
(prior participation in the major school 
reform programs in the district), grade level 
(elementary/middle/high), and region (the 
district is broken into five regions). In each 
school, classrooms are randomly sampled to 
provide a fair representation of particular 
grade levels and subject areas in each 
school. Common protocols are used so that 
data collection at each school is as uniform 
as possible. Data collectors are trained in 
advance of the snapshots and conduct their 
data collection in teams so that assessments 
of implementation reflect consensus among 
team members. 

• Snapshot results are timely. Each snapshot 
team enters their results onto the district’s 
website. The results are then aggregated in 
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time for discussion at the next monthly 
principals’ meeting. 

• Snapshots are developed and interpreted 
collaboratively. The snapshot rubrics are 
developed and refined with wide input. The 
snapshot results are discussed and explored 
for meaning and implications each month by 
school and district leaders, rather than 
simply reported. This structure and process 
is designed to reinforce a district-wide 
learning community that focuses 
conversation on issues of supporting and 
improving instruction. 

• Snapshots focus on intermediary outcomes. 
While many monitoring systems emphasize 
impacts on student test results, this system 
deliberately focuses on the link between a 
reform initiative and student outcomes, 
which consequently may shed light on the 
success of professional development 
designed to roll out reform initiatives. 
Through it, the district seeks to understand 
the degree of implementation of the 
district’s reform efforts as an important 
complement to examining impacts on 
student test performance. 

• Snapshots facilitate the building of a 
district-wide learning community. The 
design, implementation, and analysis of the 
monthly snapshots constitute a collaborative 
process that involves members from all 
levels of the district in substantive 
conversations around the core processes of 
schooling (teaching and learning) and the 
necessary structures to support these core 
processes. 

These attributes of the snapshot system 
distinguish it from other monitoring systems. 

The snapshots formalize and build upon a 
technique increasingly used in American 
education in which school leaders walk through 
a school to observe implementation and provide 
feedback to teachers and inform support 
strategies. These semi-structured observations 
are called “focus walks” or “walk-throughs” by 
the America’s Choice school reform design. 



Unlike the monitoring approach advocated by 
Leithwood and Aitken ( 1 995), which relies 
heavily on prefabricated surveys, the data 
gathered during the snapshots are primarily 
direct observational data where some concrete 
evidence (artifacts) is required to make a 
judgment about the implementation of a 
particular instructional or instructional support 
activity. The information gathered in the 
snapshots is thus likely to be more accurate and 
useful, assuming adequate training of the data 
collectors. They are in the classrooms and 
schools directly observing and making 
evaluative judgments about the quality of the 
instruction and instructional supports as they 
actually exist rather than as reported by teachers. 

However, unlike the monitoring system 
proposed by Leithwood and Aitken (1995), the 
snapshot system is not comprehensive. Rather, it 
is targeted and very specific, especially in terms 
of the specific artifacts, behaviors, and responses 
that comprise quality in the specific area being 
assessed. Finally, the snapshots are more 
contextualized than other district monitoring 
systems. Leithwood and Aitken and 
organizations like the National Society for the 
Study of Education (NSSE) have developed 
extensive and well thought-out questionnaires 
for district use, whereas the snapshot system is 
internally developed on the basis of priorities 
and goals of the district. By using prefabricated 
instruments, an important learning process for 
the district and its participants may be lost or at 
least curtailed. 

Study Design, Data 
Sources, and Analysis 
Methods 

The data upon which this article is built come 
from three sources. The first data source is 
intimate knowledge about the system and access 
to all documentation related to its development. 
The authors of this article have been providing 
evaluation and technical assistance services to 
Duval County since 1 999 and were involved in 
the development of the snapshot system. In this 



6 



Monitoring instructional Reform 



capacity they are thoroughly knowledgeable 
about the snapshot system and how it was 
developed, having had access to both the 
documentation about the system and attributes 
that were considered and then rejected. 

The second data source is the results of a series 
of formal interviews with a sample of both the 
data collectors (both principals and district 
administrators) and the principals of schools that 
were visited (called host principals). The sample 
of host principals was chosen to represent the 
range of schools in the district, based on the 
following criteria: school level, reform 
experience of the school, region within the 
district, and the state assigned school 
performance grade. During the 2002-2003 
school year, the authors of this paper conducted 
a series of telephone interviews with samples of 
both the data collectors (eight interviews) and 
host principals of schools that were visited (eight 
interviews). Using structured interview 
protocols, data collectors were asked about their 
training, their perceptions of the snapshot system 
overall, and their most recent snapshot 
experience; what they learned from their 
experiences; how the system may have 
influenced their own work; how the snapshot fit 
with other district initiatives; and their views 
about what could be improved. Host principals 
were asked about their perception of the 
snapshot system and the qualification of the data 
collectors, including their perception of the 
accuracy of the assessment, what they learned 
from their experience, what they planned to do 
with what they learned, and how they had 
prepared for the visit. All interviews were tape- 
recorded and transcribed. They were then 
analyzed to identify emerging themes, formally 
coded, and reanalyzed within each theme to 
produce the categories of this paper. 

The third data source is the results of a survey 
that was administered to all principals in the 
district who attended a monthly principals’ 
meeting in April, 2003. Of the 149 principals in 
Duval County, 126 completed a survey for a 
response rate of 85%. All principals were asked 
about their experience with the snapshot system 
(all schools in the district had been visited at 
least once by a snapshot team); how useful the 



snapshots were to them and their faculty; 
whether and how they made use of the snapshot 
rubrics; and the extent to which the resulting 
data and discussions in the monthly principals 
meetings were informative. An additional set of 
questions was included for those principals who 
were also snapshot data collectors. The survey 
data were analyzed using statistical analysis 
software, producing simple frequencies and chi- 
square analyses of significant differences over 
time and between relevant groups. 

Influence of the 
Snapshot System 

The development of the snapshot system began 
with the search for an answer to district leaders’ 
question: How do we know to what extent 
teachers and school leaders are implementing 
our reform ideas? To answer this question, the 
snapshots began with the relatively modest goal 
(which was ambitious enough in its own right!) 
of producing valid pictures at particular points in 
time (hence snapshots) of the implementation of 
district-wide reform initiatives. But as the 
system began to take shape, and as we began 
talking to educators about its impacts, we 
discovered that its influences were deeper and 
wider ranging than originally envisioned. As one 
principal told us in the spring of 2003, “Every 
meeting I go to this year, I hear something about 
snapshot and I hear the word being used in all 
kinds of ways too. I don’t thi nk there’s anyone 
who doesn’t know about it.” In order to better 
understand the uses and influences of the 
snapshots on practices in the district, we began 
to systematically collect information in the 
winter of 2002 and spring of 2003, the first year 
of the snapshot’s implementation. 

In this section of the paper, we trace some of the 
ripples that the snapshot system has made in the 
district’s pond. Many of these themes emerged 
only after we had interviewed both data 
collectors and principals whose schools were 
visited. First we discuss how the snapshots have 
provided a picture of the extent to which the key 
district reforms have been implemented within 



7 



Monitoring instructional Reform 



Supovitz and Weathers 



schools and classrooms. Second, we consider 
ways in which the snapshots have more fully 
articulated a picture of implementation of 
district reforms to principals and school 
faculties. Third, we examine how the snapshots 
have signaled district priorities to school leaders 
and faculties. Fourth, we examine individuals’ 
feelings of accountability associated with a 
system that was designed as a district-level 
accountability device. Fifth, we discuss the ways 
that the snapshot has become a mechanism for 
non-traditional professional development for the 
snapshot data collectors. Sixth, we examine how 
the snapshot experiences have created a 
mechanism for the cross-pollination of ideas 
across the district. Seventh, we discuss ways in 
which the snapshot system has become a catalyst 
for creative local uses of the snapshot concept. 
Finally, we discuss some of the important 
challenges and trade-offs that have arisen during 
the snapshot development process. The paper 
concludes with a brief discussion of the 
implications of building formal systems for 
monitoring instructional improvement reform in 
educational organizations. 

Providing a Picture of 
impiementation of Key District 
Reforms 

Most directly, the snapshot results provide a 
picture at a particular point in time of what 
implementation of a specific element of the 
district’s standards-based reform looks like. The 
results of individual school snapshots, focused 
on a particular element of the district’s 
Framework for the Implementation of Standards, 
are entered on a secure website and are then 
aggregated to form a picture of district- wide 
implementation. The results are then reproduced 
for analysis and discussion by district leaders 
and for dissemination and discussion at the 
following month’s district- wide principals’ 
meeting (see Appendix C for an example of 
results). 

Comments from district administrators and 
principals suggest that the snapshot results are 
focusing attention on implementation of the 
district’s reform efforts. As one district 



administrator said, “I think it is forcing us to 
take a look. We have been talking about 
standards. Okay, now here is an opportunity to 
actually take a look at it and determine where we 
are on this road towards standards.” 

District leaders also report that the snapshot 
results give them a better understanding of the 
extent to which the district’s reform ideas are 
being practiced in schools and classrooms. 

Initial snapshots revealed, for example, that the 
implementation of some reform efforts has not 
taken place to the degree that district leaders 
would like. “We talk about standards, safety 
nets, and student work all the time, but I am 
always a little surprised to learn how 
superficially they are understood and used in 
some schools,” said one regional superintendent. 
Principals also reported that they found the 
snapshot results to be useful for gaining a clearer 
picture of the level of implementation in their 
own schools. On our spring 2003 survey, for 
example, 84% of the principals agreed or 
strongly agreed that the snapshot system has 
provided the district with valuable information 
about the implementation of standards. 

Respondents also noted the shift of emphasis 
towards implementation, as opposed to the final 
impact of programs. “[The superintendent] is 
trying to find something that will give him an 
idea how we are implementing the standards- 
based design,” one principal said about the 
snapshot system. Another district administrator 
highlighted the value of the intermediary data 
provided by the snapshot results. “We’ve 
consistently said that you have to do monitoring 
to determine how things are going. You don’t 
wait until the end of a grading period to 
determine that 40% of the ninth graders are 
going to have failing grades. I think that this 
reinforces what we have been saying in the 
district. I think all of our initiatives are tied back 
into academic performance and how you 
monitor that and how you take those quick shots 
of it to determine if are we moving in the right 
direction or do we need a course correction,” he 
explained. By focusing on implementation, the 
snapshot results give district administrators and 
school leaders information with which to guide 
their efforts to deepen implementation of the 
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district’s reform efforts in order to increase the 
likelihood that these efforts will translate into 
higher student test performance at the end of the 
school year. 

Articulating Deeper 
Implementation 

A reform vision provides a crucial articulation of 
the direction in which an organization seeks to 
head. Yet a vision is rarely specified enough to 
provide members of the organization with clear 
direction (Gillen, 2000). Louis and Miles (1990) 
argue that vision emerges from collective 
reflection on action, rather than from mere 
vision statements. The snapshots have provided 
a deepening understanding of the key 
components of the district’s reforms. For 
example, one snapshot data collector, a middle 
school principal, talked about how the snapshots 
have helped her to distinguish between the 
different levels of understanding that students 
have about standards. She also realized that 
teachers may not be adequately helping students 
to make connections between their work and 
expectations for their work, as evidenced in the 
standards. 

It [a snapshot visit] has alerted me to how a 
lot of teachers will put the standard up 
there, but the kids have no idea the 
standard’s there or what it means or that the 
lesson is tied to it. I guess I should have 
known that, but it made me see that now. I 
ask my kids when I walk into the class, 

“Hey, do you have a clue what standard 
you ’re working on? ” Amazingly, they don ’t. 
Some of the things that I noticed when I was 
in this school, the kid would say “Yeah, 
we ’re working on that standard over there 
that the green finger is pointing to. ’’ They 
couldn ’t tell me what the standard actually 
said or what it was about. They just knew 
that they were working to the one that the 
green finger was pointing on, because their 
teacher always moved the green pointer to 
the standard they were working on. So they 
really didn ’t have any substantial 
understanding. 



The development of the snapshot rubrics 
provided crucial articulation of different degrees 
of implementation and provided teachers and 
school leaders with more concrete specificity as 
to what different levels of implementation of the 
district’s reform vision looked like. Principals 
overwhelmingly indicated that the snapshot 
rubrics have provided greater specification of 
the district’s instructional vision. On the spring 
2003 survey, 41% of principals strongly agreed 
and 49% somewhat agreed with the statement 
“The snapshot rubrics have helped me to better 
understand what to focus on in order to 
implement standards-based reform.” 

In interviews, the district administrators in 
charge of regions within the district talked about 
how they used the snapshot rubrics for 
discussions with principals and how they 
provided greater specificity for principals to 
guide them in their implementation. “I can see 
me using it for discussions with principals, too, 
and for collaboration,” said one regional 
director. “I think it certainly lays it out for 
principals,” said a regional superintendent. 

They determine where they are, but through 
the rubric, they can see where they need to 
go, and so can the teachers, and so can I 
and the district, so that we can see what else 
needs to be happening to help them to get to 
all the way over there to In Place. We know 
it ’s going to take a while; it ’s not going to 
happen in a year or two. But it gives them a 
road map, and I think this is something 
principals had really wanted — what are the 
expectations. 

One of the key determinants to successful 
change within any large organization is the 
extent to which the language of the reform is 
shared across members of the organization (Daft 
& Huber, 1987). By developing and spreading a 
particular reform language, district leaders are 
also giving teachers and school leaders a 
vocabulary with which to develop a co mm on 
meaning and deeper understanding of reform. 
Developing a common vocabulary with which to 
discuss current and desired practice is a large 
reform step in itself because it is a foundation 
for problem-solving interactions. 
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In our interviews, several snapshot partieipants 
deseribed how the snapshots had provided a 
eommon language with whieh to engage eaeh 
other about the distriet’s reforms. One distriet 
administrator, for example, diseussed how the 
snapshots provided a meehanism for spreading 
what he called a “common platform” across the 
district. As he explained: 

It certainly now gives a common platform 
for discussion. For example, the next piece 
dealing with safety nets. Now we have been 
talking about safety nets and encouraging 
schools to do safety nets, but now I think we 
can take those particular questions 
(referring to rubrics and evidence forms) as 
I work with and visit other schools to talk 
about safety nets. This allows us to have the 
same kind of dialogue and conversation with 
principals across our region. I think that ’s 
very helpful to me. Now we have the same 
platform and I’m not on one platform with 
one principal and on another platform with 
another. 

Particularly given the size of Duval County (149 
schools, 7300+ teachers), developing a shared 
understanding both within and across schools is 
particularly challenging. And yet, this is what 
the snapshot process seems to have facilitated. 

The snapshot rubrics in particular have created a 
means of disseminating a common language of 
the district’s reforms across the schools in Duval 
County. As one district administrator 
commented: 

The snapshot rubrics have been very helpful 
just as talking points when I go talk to 
administrators, principals. And I think 
they ’ve helped teachers. It ’s been helpful to 
teachers as any rubric would... be in 
learning of what we ’re looking for. It has 
given us a common theme, a common goal — 
don ’t know if it ’s a goal — a focus 
everybody’s on... Everybody’s got this 
common focus now. 

Signaling District Priorities 



School districts are distinctly nested 
communities, in which school faculties tend to 
operate largely within their own insulated micro- 
societies (Orton & Weick, 1990; Weick, 1976). 
School principals spend most of their time and 
energy operating within, managing, and 
responding to the particular issues of their own 
school communities. Within this din, the 
snapshot activities act as a reminder to principals 
and teachers of the larger framework within 
which they are operating and of the priorities of 
district leaders. One elementary school principal, 
for example, talked about the how the snapshot 
rubric on Understanding and Using Standards 
kept her school focused on standards. “It kinda 
keeps it [standards] there at the top of your list. 

It pushes it back up the list because you see it is 
still important to the district,” she said. Others 
saw the snapshots as an extension of the 
superintendent’s priorities and his commitment 
to staying the course and accomplishing the 
goals he set as priorities. As one principal 
commented, “One by-product of this is that 
people know that he is totally serious. It’s 
[district reform efforts] not going away and 
we’re still going to do this and it’s going to 
remain a priority.” The snapshots thus were a 
way for district leaders to penetrate protected 
school cultures in order to communicate the 
larger district goals and priorities to school 
faculties. 

Another prevailing theme that emerged from our 
interviews with principals and administrators 
was the sense of urgency to implement the 
district initiatives that was conveyed by the 
snapshot activities. District leaders were actually 
examining what teachers and principals were 
doing and expecting them to be practicing in 
ways prescribed within the snapshot rubrics. As 
one middle school principal said about enacting 
the snapshots, “I think this is one of the most 
powerful things that he [the superintendent] has 
done. It is creating a sense of urgency, because 
people know that they are going to be visited 
and they want to do their best.” An elementary 
school principal reiterated this theme, “Just 
knowing that someone else from the district 
could come into my school and do a snapshot of 
my faculty and staff has created a sense of 
urgency. So I feel I like I need make sure that all 
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my people are prepared. . One regional 
superintendent explained his pereeption of the 
effect of the snapshots on principals’ attention to 
standards: 

/ think for the first time, we ’ve got principals 
that have started paying attention to it 
[standards] . I think some maybe thought 
that this too was going to go away. But I 
think once you start creating a vehicle that 
is going to be consistent in the monitoring 
piece, people then respond because they 
know this is important because it ’s being 
monitored: an inspection is taking place. So 
1 now give this a higher priority as to my 
management attention. 

Thus the snapshots raised the level of awareness 
of and attention to the district initiatives. The 
snapshots focused school personnel on the 
district’s standards-based reform initiatives. The 
snapshots thus signaled the district’s priorities to 
school leaders and faculty members. 

Similarly, teachers sometimes have difficulty 
distinguishing between reforms advocated by 
their principals and reforms advocated by 
district leaders. The snapshots lent greater 
weight to the reforms that principals were 
advocating and gave them a way to convince 
their faculty that the reform s were not just local 
to their schools but were occurring district-wide. 
As one principal noted, “It is encouraging the 
teachers to adopt standards if they were 
borderline or if they were waiting for it to go 
away. They are seeing concrete evidence that 
it’s real, that it’s not just my principal and it’s 
not going to go away too soon.” A district leader 
made a similar point about the value of the 
district making clear its priorities through the 
implementation of the snapshot for aiding the 
work of principals in furthering the work of 
getting teachers to embrace standards-based 
instruction. She explained: 

It [the snapshot] is a very useful tool as a 
driver to help people adopt it more quickly 
than what some people would like to. Some 
people would like to wait until it goes away. 
But some principals have really struggled 



with how to get their teachers to adopt this, 
and so when we started giving out the 
rubrics, then the principal could say it ’s not 
]ust me, this is really here. So it gave the 
principal a tool to use and it could be used 
very effectively. . . 

Providing indirect 
Accountabiiity 

The snapshot system is explicitly designed to 
measure the depth of implementation of the 
district’s initiatives. In this sense it is a district- 
level accountability system, because the unit of 
interest is the entire district, not individual 
schools. Schools are data points to form a global 
analysis. All design elements of the snapshot 
system reinforce this point. Schools are selected 
as a group to broadly represent the range of 
school types (elementary, middle, and high), 
regions within the county, and state test 
performance, such that together they represent 
the swath of the district. Teachers within schools 
are sampled to represent the range of 
implementation within the school. Although 
everyone knows which schools are in each 
sample, only the snapshot data collectors and the 
principal of the school kn ow what rating the 
school received. This anonymity is preserved 
when the results presented at each monthly 
principals’ meeting. Shared results show the 
global picture of implementation across the 
district and individual schools, but individual 
schools are never identified either publicly or to 
district administrators. 

The identity of the results of individual schools 
is closely guarded, both publicly and privately. 
Ever since its introduction, public 
pronouncements continually reinforce that the 
results for individual schools will never be 
identified and that the purpose is to develop a 
picture of district-wide implementation. The 
superintendent and the system developers have 
continually stressed this point at every 
opportunity. In one famous debriefing session 
for one of the early sets of snapshot results with 
the district’s instructional cabinet (the 
superintendent, chief of staff, and five regional 
superintendents) led by the first author of this 
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report, the superintendent wanted to know the 
identity of one of the high-performing outliers in 
order to honor the school. The first author 
explained, “Well, I could tell you that but I am 
not going to, because if I tell you the identity of 
that school, the next thing you would want to 
know is who is the lowest performing school 
and the moment that you identify any schools 
publicly then the system is going to die from 
suspicion and mistrust.” And so we made a pact 
that the purpose of the system was to get the big 
picture of the district and that no individual 
schools would be identified, even behind closed 
doors. 

But organizational suspicions die hard. Despite 
the consistent pronouncements to the contrary 
(although there were cases of gossiping by data 
collectors in which they revealed their findings 
to others, thus undermining trust in the 
confidentiality of individual schools’ results), 
principals still felt that they were being 
individually held accountable for their 
performance on the snapshots. In the spring 
2003 survey, principals were asked to respond to 
the statement: “The snapshot results for my 
school are used by district administrators to 
make judgments about my individual school’s 



implementation.” Overall, 56% of the principals 
agreed with this statement! Figure 1 shows the 
results decomposed by whether or not the 
principal participated as a snapshot data 
collector. 

Over 60% of the visited principals strongly or 
somewhat agreed that the snapshot data was 
being used by district administrators to judge 
their individual school, despite all the 
proclamations by district administrators to the 
contrary. These principals were significantly 
more likely to believe this than the principals 
who were snapshot data collectors, of which 
about a third somewhat or strongly agreed this 
was the case. Principals who were data 
collectors were likely less suspicious of district 
administrators, due to their greater familiarity 
with the goals and procedures of the snapshot 
system. But surprisingly, even a third of these 
principals still suspected that administrators 
were making judgments about their individual 
school’s implementation, suggesting that the 
district administration had not adequately 
dispelled principals’ fears of the misuse of the 
system. 



Figure 1. Responses to the statement “The snapshot results for my school are used by district 
administrators to make judgments about my individual school’s implementation.” 



100 

80 

60 



□ strongly agree 

□ somewhat agree 



40 

20 

0 



33 



10 



23 



Snapshot data collector 



64 




12 



Monitoring instructional Reform 



Any system that monitors implementation of 
reform programs puts pressure on partieipants 
and is likely to make them feel aeeountable for 
their eomplianee. The mere presence of the 
system reminds teachers and principals that 
these are elements of particular concern to the 
district. The designers of the snapshot system 
sought to soften this pressure (and the associated 
anxiety and resistance) by making the district the 
unit of accountability and protecting the identity 
of individual schools. Even so, principals’ 
responses to survey and interview questions 
revealed their feelings of individual 
accountability. 

Principals also reported that they felt 
accountable for their school’s implementation of 
the content of the snapshots. On the spring 2003 
survey of school principals, 47% strongly 
agreed, and an additional 40% somewhat agreed 
to the statement “We are held accountable for 
our school’s implementation of the content of 
the snapshots.” This agreement likely reflects 
the tight relationship between the snapshot 
subjects and the emphases of the district’s major 
initiatives. This sense of accountability also 
probably stems from normative pressures that 
accompany increased interaction between staff 
which lays open to public judgment what is 
going on inside the walls of a school. 
Professionalism also may likely play some role, 
as one district administrator pointed out to 
explain how the snapshot has played a powerful 
role in generating a sense of urgency and focus 
around district key district reforms. As the 
principal explained, “So somebody comes to 
your school, or somebody may come to your 
school, all of a sudden . . . you want to do your 
best for the superintendent and for your kids.” 
This pressure appears to be greater for data 
collectors, who see themselves in a position as 
exemplars. This awareness is described by one 
data collector principal: “And so how can I 
expect to go and see these things in other 
schools when I had the benefit of all this 
training, and I’m not sure I’ve seen it in mine? I 
need to make sure that it is happening here. It’s a 
reality check for me.” 



Supplying Professional 
Development Opportunities 

Adults learn in so many different ways and in so 
many different situations. However, we tend to 
see their learning opportunities as limited to 
formal professional development occasions, as 
opposed to non- canonical, yet powerful, learning 
experiences (Brown & Duguid, 1996). One of 
the surprising things that emerged from our 
interviews with the snapshot data collectors was 
the extent to which the entire snapshot 
experience - from the scrutiny of the snapshot 
rubrics to the visits to schools with colleagues to 
the examination of the snapshot results - 
provided a powerful learning opportunity for 
those involved. 

The training sessions for new rubrics became 
important professional development 
opportunities for the snapshot data collectors. In 
advance of using a rubric new to the district, the 
data collectors would get together, review the 
rubrics and evidence forms, break out to practice 
their application in a small group of 3-4 schools, 
and then reconvene to compare ratings and 
discuss how the snapshots went and make 
suggestions for fine-tuning the rubrics and 
evidence forms. Many of the data collectors 
lauded the value of the training sessions and 
how they deepened their understanding of the 
snapshot topics. “The training sessions really 
helped me to understand what the district was 
trying to get at with its emphasis on standards 
and student work,” said one snapshot data 
collector, and elementary school principal. “I 
never thought of this [the snapshots] as 
professional development, but I found those 
trainings so much more valuable than the other 
district training sessions,” said a middle school 
principal. 

Cross-school visitations provided valuable 
learning opportunities for school and district 
leaders. Not only did they get to share ideas 
about how to help improve instruction in their 
schools or regions, but these visits also provided 
direct observation of different approaches to 
teaching and learning that served as points of 
comparison and reflection for generating ideas 
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on how to improve one’s own school. One 
principal described the impacts of the data 
collection visits: 

You can come back to your building after 
you ’ve done a visitation like that and you 
learn from that experience, and so okay, I 
know I need to do this or don ’t need to do 
that, or I’ve not done that very well... But 
certainly you come back and you do 
realignments — that ’s what the process is 
about. I was glad to I was able to do that, 
because it helps me make a better 
assessment in terms of what I need to do 
with my staff. 

In our debriefing sessions with snapshot data 
collectors, we frequently heard similar feedback 
about what a powerful learning experience the 
visits were for them. 

Both the influence of the snapshot experience 
and its perceived utility was amplified for those 
principals who were involved in conducting 
snapshots in comparison to those who just were 
recipients of the snapshots. Participation in 
evaluation activities has been shown to amplify 
the influence of the evaluation, particularly its 
findings, on the learning of participants. Cousins 
(1998) examined the impact of an evaluation on 
the learning of educators at different levels of 
involvement with the data collection and 
analysis process. He found “different patterns of 
influence on organizational learning at different 
levels within the organization,” with those closer 
to the project demonstrating more learning 
relative to those who were just consumers of the 
results of the project (p. 145). 

Comparisons of the survey responses indicated 
that the principals who participated in collecting 
the snapshot data were more influenced by the 
snapshot experience. Table 1 shows the 
responses to survey items of principals who 
were snapshot data collectors compared to those 
whose schools were visited, and indicates 
whether these differences were statistically 
significant. The principals who were data 
collectors reported that the different components 
of the snapshot system (examining the rubrics, 
the visit itself, and examination of the results) 
had significantly more influence on themselves 



and their faculty than did the tools and 
experience for the principals who were merely 
peripheral participants. This is not to say that the 
visited principals did not report influence from 
the experience, only that it was more profound 
for those with a deeper understanding of the 
system and its purposes. This is one of the 
reasons that the designers of the system chose to 
more than double the group of data collectors 
from 42 in the first year to 86 in the second year, 
and to a planned 191 participants in the third 
year. 

As district leaders started to regularly examine 
the snapshot results in their own meetings, they 
began to see implications for district 
professional development and sought to more 
closely align the findings of the snapshots with 
the sequence of both principal leadership 
training and teacher training. One district 
administrator, for example, described how the 
snapshot results have increased the coherence of 
the district’s training designs. She said that the 
results give “information for what we need to do 
as far as training, and I guess the biggest part is 
our training has not been as cohesive as it needs 
to be.” In the second year of implementation, the 
district’s leadership professional developers 
began to align the topics of the snapshots with 
leadership training such that snapshots were 
conducted one to two months after topics were 
examined and discussed in professional 
development. Thus the snapshots acted as 
follow-up training and implementation for 
district reform strategies. 

The snapshot results also influenced professional 
development decisions at other levels of the 
district. A few principals told us that they used 
the results of the snapshot visits to their schools 
to guide their within-school professional 
development decisions. “It [the snapshot system] 
is also creating or identifying the need for 
training,” an elementary school principal told us. 
“A number of people haven’t adopted 
[standards] because they need systematic 
training and assistance in getting there,” she 
said. School coaches also reported that they 
were examining snapshot results for indications 
as to further emphases they could make in their 
training with teachers in their schools. 
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Table 1. Principal Reports of Influence of the Snapshots on Themselves and Their Faculty 





Visited Principals 
(n = 93) 


Principals who were Data 
Collectors 
{n = 33) 


Survey Item 


Somewhat 

Agree 


Strongly 

Agree 


Somewhat 

Agree 


Strongly 

Agree 


1 have disseminated the snapshot 
rubrics to my faculty * 


30 


62 


7 


90 


The snapshot visit to my school was 
useful to my faculty * 


53 


26 


30 


53 


1 have examined the snapshot rubrics 
for topics other than the one on which 
my school was visited.* 


33 


56 


9 


84 


The discussions of the snapshot 
results at the monthly principal 
meetings provide me with guidance as 
to what to focus on in my school.- 


52 


21 


29 


42 


1 regularly share the snapshot results 
with my faculty. 


43 


25 


38 


41 



* p < .05 ~ p < .10 

(* Statisticaiiy significant differences between principals who were data coiiectors and visited principals) 



Cross-Pollinating ideas 

Another theme that emerged as we examined the 
data from our interviews was the extent to which 
the snapshot became a means for the sharing of 
ideas across the district. The discussions of the 
snapshot results at the district’s monthly 
principal’s meetings was designed to facilitate 
conversations rather than to simply present 
results to principals. The district’s five regional 
directors are coached to facilitate an 
examination of the results with principals in 
their region in order to collaboratively arrive at 
conclusions about the meaning of the results and 
to brainstorm actions based upon their findings. 
Several respondents noticed the contrast 
between these conversations and many other 
meetings. As one principal explained: 

We go to meetings ... I go to meetings with 
other principals all the time. We are given a 
lot of information. We sit in meetings and we 
get a lot of information. Other meetings, 
we ’re required to do this and that. There are 
very few times when we really sit and talk to 
another professionally about what ’s 



happening in our school and what ’s 
working. ‘What do you say to teachers who 
do this? ’ I think it’s really powerful. . . And 
just hearing another person talk about the 
words they use in conversations with 
teachers helps. So that to me has been a 
powerful part of this. It has given 
opportunities for administrators to really 
talk about best practices and what they 
could do to help their school improve. 

A similar sense of dialogue as opposed to 
dissemination arose in conversations about the 
debriefing experience between the host principal 
and snapshot visitors that concluded snapshot 
visits. One snapshot data collector, a district 
regional director, said: 

/ think it is especially valuable to hear 
conversations between principals from 
different regions about what they saw and 
what their understanding was and what 
their training has been and how they have 
accomplished things, especially with moving 
their teachers, because I think really the 
most difficult part is not the presentation of 
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information, it ’s having staff to adopt it, and 
how do you get that to occur. So it ’s been 
helpful to see principals exchange 
information. 

Similarly, a district administrator observed how 
the snapshots helped to strengthen and validate 
the professional judgment of participating 
principals: 

Another thing I learned is that when I met 
with the administrators after the 
observations, those were very powerful for 
the principal. I think those meetings give the 
principals words to say, another way of 
saying what they already may know, 
because when another administrator comes 
in and sees things that this principal has 
already seen and then says it in a different 
way. It ’s so strengthening to the principal to 
say, “Oh, they see what I see, but they ’re 
telling me a way they ’re seeing it that 
sounds different from what I’ve always 
said. ’’ 

These debriefing sessions between the data 
collectors and the principal of the visited schools 
were a catalyst for professional conversations. 
One district administrator said: 

/ noticed there was a lot of talk about best 
practices between the administrators, what ’s 
working what’s not working, what I’ve tried. 
They get off subject of the snapshot a little 
bit because they ’re saying, “Well, how did 
you get them to do that? What do you say 
when they say this? ” A lot of powerful talk 
goes on in those that ’s not really about the 
snapshot. It creates a moment for more 
collegial conversations I guess ... 

The snapshots are designed to bring together 
principals who are at different developmental 
levels in a mutually constructive learning and 
teaching experience. While it is possible for less 
experienced principals, or principals of schools 
that have not deeply implemented the district’s 
reform efforts, to be in a position of conducting 
snapshots in schools with more experience, the 
design of the snapshots mitigates these situations 
via several design elements. First, the district 



leaders are careful to choose principals to 
conduct the snapshots who are generally 
considered strong leaders. Further, data 
collectors visit schools in teams of two or three 
(depending on the school’s size), which reduces 
the possibility that the entire snapshot judgment 
will be made by a single individual (and 
increases the opportunities for learning). Finally, 
when putting together the teams for each visit, 
the assignors of the snapshot teams are careful to 
distribute knowledge and experience. 

Principals reported that they learned from their 
peers with deeper experience. One regional 
director explained, “The most positive thing that 
happened was they have connected with a school 
that has two or three more years of training, and 
they will visit and do some working together. So 
I think it was a very positive thing to connect 
one school with the other for them to support 
and help each other.’’ An elementary school 
principal saw the experience of visiting another 
school as tremendously beneficial in terms of 
gathering new ideas. The principal explained: 

Just using the information that I see at other 
schools and bringing it back and 
incorporating it, I was able to talk with the 
principal there about how much further 
along I feel that her school is than this 
school, and maybe just get some ideas like 
her training about standards, how she did 
follow-ups in the classroom, and what 
observation tools she uses with her teachers 
when she goes in there to give them vital 
feedback on how they ’re progressing. 

Principals also used their visits to compare their 
schools with other schools. As one high school 
principal said: 

So that ’s [data collection visits] made me 
better, better prepared, as I said. I hate to 
use the word compare, but I certainly can 
come back and see where we are compared 
to somebody else. When I went to the first 
school and the training and I watched that 
unfold, I felt that I had a lot of work to do at 
my school, because they seemed to be a lot 
further along than we were. But then when I 
came back and had my debrief with the 
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snapshot people, then I feel like, maybe 
we ’re not as bad as I thought; you know 
sometimes you ’re too critical of oneself. But 
then I went to the snapshot visit myself and 
came back, then I thought maybe we ’re 
further along than they are. You can ’t help 
but compare; it ’s just in the process of what 
we do, and I think that ’s okay. 

Spawning a Variety of Locai 
Uses 

One testimonial to the utility of both the 
snapshot eoneept and the tools created by the 
snapshot developers is the variety of ways that 
the snapshots are being modified and used at 
different levels of the organization. Principals 
and district leaders reported that they have found 
ways to take the snapshot rubrics and the 
systematic nature of data collection inherent in 
the snapshots and apply these concepts to other 
circumstances. 

Across the district, for example, principals 
described how they are using the snapshot 
rubrics in their classroom observations. “Its 
helped me with my own school because I’m 
actually looking with greater depth when I get 
into classrooms and giving the teachers feedback 
based on that,” one of the snapshot data 
collectors told us. Another principal who we 
interviewed, not one of the snapshot data 
collectors, indicated that she had also integrated 
the snapshot rubric into her observations. She 
said: 

/ have taken the rubric and I’ve 
incorporated that into an observation form, 
so I’m looking at the same things in the 
classrooms as the snapshot visitors. I make 
my recordings on that classroom 
observation form and leave a copy with the 
teacher so that they have feedback from me, 
so that I’m looking for the same things that 
the district thinks is important to look at. 

Principals were also using the rubrics with their 
staff as the basis for conversations about 
implementation and improving the effectiveness 
of district reform policies within their own 



schools. “I recently used the safety nets rubric in 
a faculty meeting so that we could discuss and 
improve the way we were providing safety net 
support to our students,” said one elementary 
school principal. These conversations between 
principals and teachers are essential for ensuring 
deeper understanding and implementation of 
district reform efforts, and the snapshot has 
clearly been a catalyst in encouraging them. 
Another principal further confirmed this point 
when he noted that “As I said, we talked about 
the snapshot visit, we sat and we had a meeting, 
and we debriefed, and they’re talking the rubric 
and the standards . . . and I got some feedback 
from them. Not doing the snapshot, I wouldn’t 
have gotten that information.” 

Principals were also interested in taking 
advantage of the expertise of the snapshot data 
collectors. Several of the snapshot data 
collectors have been asked to visit other schools 
to conduct informal assessments based on the 
snapshot rubrics. As one of the data collectors 
explained: 

People have called me and asked me to 
come and visit their school, even when they 
know they weren ’t chosen. They just want to 
know what it looks like, you know, how are 
we doing? Do you have some advice for us? 
What should be our next step in 
implementation ? 

This suggests that schools are eager to 
implement the district’s reforms, but are 
uncertain about their own levels of 
implementation and are seeking guidance as to 
how to move forward. 

In another interesting offshoot, principals are 
including the snapshot rubrics into their self- 
constructed performance evaluation system, 
called Appraisal Plus, which requires them to 
identify their own performance goals and 
measure their success at achieving those goals. 
One of the regional superintendents we 
interviewed reported that more than a third of 
the principals in his region were using the 
snapshots as one measure of teacher progress 
within their school. He explained, “They just 
saw this as a tool that was already developed. . . 
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It already has some credibility. You know, it has 
the district's stamp of approval, and all these 
principals have already gone out and used it.” 

Challenges 

The introduction and ongoing use of snapshots 
has also raised a series of challenges for the 
snapshot developers. Here we discuss some of 
the key challenges that both designers and 
participants have pointed out. These include 
rubric development, which necessitated reaching 
consensus on greater detail of what 
implementation meant; maintaining the integrity 
of the system as the pool of data collectors 
expanded; and capturing the learning provided 
by the snapshots. 

One of the hardest challenges is developing the 
rubrics that are to be used in the snapshots. The 
rubrics are so important because they are the 
embodiments of the district’s definitions of 
high-quality implementation and specify the 
stages that teachers and schools go through in 
deepening their implementation of the district’s 
vision of standards-based instruction. In the first 
year of the snapshot development, there was one 
group of rubric developers, made up of 
principals, regional directors, and technical 
assistors who developed the rubrics, regardless 
of topic. The rubrics were then vetted by larger 
circles of school and district leaders. This 
process was sufficient for the topics of the first 
year, which were: Understanding and Using 
Standards, Connecting Student Work to 
Standards, and Safety Nets. 

In the second year, the snapshot topics became 
more ambitious. Data-Driven Decision Making, 
Connecting Reading Instruction to Standards, 
and Connecting Mathematics Instruction to 
Standards were added as areas for snapshot 
inquiries. These rubrics proved to be more 
difficult to develop and attain consensus 
agreement on their contents. The development 
teams became more specialized, bringing in 
content experts in these areas, and the review 
periods became longer as the development teams 
received iterative feedback and refined the 
instruments. Training the snapshot data 



collectors also became more involved, as the 
requirements for expertise in order to make the 
judgments required of the snapshots became 
greater. 

Particularly in the second year, as the teams 
delved deeper into the thickets of specifying 
instructional implementation, the rubric 
development process led to serious, and 
sometimes heated, debates about different 
conceptions of quality implementation. These 
differences were long standing across the 
district, and had led to different applications 
across the district, but nothing had heretofore 
forced them to the surface. Not surprisingly, the 
more specified the rubrics were, the more 
contentious the conversation became because we 
were rubbing up more closely against deeply 
held beliefs about what was the “right” way to 
teach. When the rubrics were more general (e.g.. 
Understanding and Using Standards), it was 
easier to reach agreement because agreement is 
easier for more general ideas. The explicitness 
and publicness of the rubrics, which are 
essentially codifying the expectations of the 
district, are helping to bring out the different 
conceptions of various stakeholders. This has led 
to important conversations about what is meant, 
for example, by reading across the curriculum, 
professional learning community, and 
preventative safety nets for at-risk students. The 
more in advance the rubrics are developed, the 
more opportunity they have to be critiqued by 
people from across the system, the more 
iterations of refinement they go through, the 
more credible they become and the more deeply 
articulated and widely shared will be the 
district’s vision for instructional quality. 

Another major challenge that the development 
teams faced was the extent to which the rubrics 
should specify the desired state of instructional 
quality, as opposed to moving the current state 
of quality forward. The deeper the teams got into 
the rubric development process, the more they 
realized that they were specifying steps along 
the path that they hoped schools and teachers 
would go down — and there is a lot of value to 
articulating what schools should strive towards. 
But they also realized that they ran the risk of 
getting out too far ahead of both where the 
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district currently was and what the data 
collectors could reasonably assess. In one 
training session on the rubric for Connecting 
Student Work to Standards in Reading, a high 
school principal commented, “You don’t even 
need to go out and do a snapshot on this rubric. I 
can tell you before you start that my school is at 
the Preparing Stage.” Other principals readily 
concurred. They were pointing out that the 
rubrics were way out ahead of where they 
currently were. 

Another challenge that the developers faced was 
how to maintain reliability as the number of data 
collectors expanded from year to year. In the 
first year of the snapshot, when there were 42 
data collectors, the project succeeded in 
maintaining a fair level of reliability and 
validity. On the spring 2003 survey of all the 
principals in the district, 84% of the principals 
agreed that the snapshot data collectors were 
qualified to assess implementation of standards 
in their school (45% strongly agreeing and 39% 
somewhat agreeing). Over three quarters felt that 
the snapshot completed of their school was a fair 
assessment of their implementation of that 
aspect of standards-based reform (45% strongly 
agreeing and 33% somewhat agreeing). 

In the second year, the data collectors expanded 
to 86, and the problems of maintaining rigorous 
and disciplined visits expanded as well. This 
was compounded by the increasing 
sophistication of the snapshot rubrics, which 
consequently required raters with higher skill 
levels. In the second year, training activities 
became more involved, with vetting of rubrics 
and practice snapshot visits to three to four 
schools. As the district planned to add assistant 
principals, vice principals and coaches to the 
snapshot teams for the 2004-2005 school year, 
assuring inter-rater reliability became a more 
pressing issue. 

Finally, it has become increasingly clear that the 
snapshots can become a way to foster 
organizational learning within the district. Yet 
within these opportunities resides a series of 
challenges. First, within an organization that is 
more used to disseminating information than 
fostering conversation to build shared meaning. 



the opportunities for school and district leaders 
to puzzle through the meaning of the snapshot 
results are often scarce and underutilized. 
Second, it is an ongoing struggle to capture the 
rich comments, insights, and learnings that do 
come from discussion of the snapshot results, 
codify them, and disseminate them to those that 
could capitalize on these insights. Thus both the 
opportunities to squeeze meaning from the 
results of the snapshots and the mechanisms to 
feed these findings back into the system are 
underdeveloped. 

Discussion 

The monitoring of instructional reform 
initiatives is a powerful, but relatively untapped, 
way of distributing common understandings of 
practice throughout large systems. Duval 
County’s experience with the snapshot system 
demonstrates how systems designed to measure 
the implementation of a district’s instructional 
reform initiative can not only provide insight 
into the depth of implementation of reform 
initiatives, but help to shape that meaning and 
become powerful organizational learning tools. 
The utility of any instructional implementation 
monitoring system is predicated on a system- 
wide vision of instruction, which can be 
reasonably expected to exist in classrooms and 
schools across an educational system. 

Design elements of any instructional monitoring 
system must be carefully considered, because 
they will determine the reliability and validity of 
the data that are produced, the learning 
opportunities embedded within the system, and 
therefore the credibility and ultimate survival of 
the system itself In this case, key design issues 
of the snapshot system included the careful 
selection and alignment of the snapshot topics, 
the selection of data collectors as broadly 
representing leadership of the system, the careful 
training of the snapshot data collectors, the 
iterative development process of the snapshot 
rubrics and evidence forms, the decision to 
provide results aggregated to the district level, 
and the opportunities to discuss the snapshot 
results. 
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As the reflections of participants in this system 
show, collaborative implementation monitoring 
systems can become powerful organizational 
learning tools. Systems in which leaders from 
across an educational system participate in the 
construction of instruments, receive training, 
collect the data itself, and mull over the results 
can deepen the buy-in and understanding of a 
district’s reform vision, facilitating deeper 
implementation. Thus a system designed to 
monitor implementation may contribute to the 
deepening of the implementation it is intended to 
capture. The snapshots have become an 
important data element in Duval County 
superintendent John Fryer’s dashboard. He is no 
longer flying blind. 
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Appendix A - Sample Rubric 
and Evidence Form 






Standards Implementation 

Snapshot System 



A system for tracking the implementation of Standards in Duval County 



Rubric: Safety Nets 

Target 2 of the Duval County Framework for Implementation of Standards asks principals and teachers to 
provide safety nets for all students. This rubric describes different levels of implementation of the safety 
net component of the target. 



Host Principal Rating: 



1 . Preparing 


2. Getting Started 


3. Moving Along 


4. In Place 


□ 


□ 


□ 


□ 



Your ratings (Mark all that apply, and then assess the overall phase of implementation): 



Preparing 


Getting Started 


Moving Along 


In Place 


□ Principal analyzes 
last year’s assessment 
data and/or beginning of 
the year baseline 
assessment data to 
identify and place 
students in need of 
remediation. 

□ School-wide action 
plan identifying specific 
students, previous 
interventions, and 
potential new safety net 
services is developed. 

□ School has some 
programs for students 
who need extra time/ 
instruction to meet 
standards, but options 
are limited. 

□ Few classroom 
teachers have identified 
at-risk students in their 
classes and have a plan 
for moving the students 
up to standard. 


□ Leadership team analyzes 
last year’s assessment data 
and/or beginning of the year 
baseline assessments to 
identify and place students in 
need of remediation. 

□ Leadership team reviews 
existing programs (i.e. before 
school, after school, 

Saturday, course recovery, 
team-up, mentoring, tutoring, 
other remediation) to see if 
data, identified students, and 
safety nets are aligned. 

□ Some classroom teachers 
have identified at-risk 
students in their classes and 
have a plan for moving the 
students up to standard. 

□ Remediation sessions are 
focused to provide intense 
instruction to move students 
closer to standards. 

□ School leaders are 
regularly monitoring student 
progress. 

□ Safety nets are organized, 
scheduled, and focused on 
critical areas for 
improvement 


□ Leadership team, as well as 
teacher teams of various 
configurations, regularly 
analyzes student performance 
data. 

□ Teachers are provided with 
training specifically on how to 
support at-risk students. 

□ School provides a variety of 
programs for students to meet 
standards (i.e. before school, 
after school, Saturday, course 
recovery, team-up, mentoring, 
tutoring, other remediation). 

□ Most classroom teachers can 
identify at-risk students in 
their classes and have a plan 
for moving the students up to 
standard. 

□ Most instructors of safety net 
courses have appropriate 
content expertise. 

□ There is a formalized 
communication system 
between safety net teachers 
and students’ regular 
classroom teachers. 

□ Teachers are provided with 
quarterly updated data on all 
students. 


□ Leadership team, as well as 
teacher teams of various 
configurations, regularly 
analyzes a variety of student 
performance data and adjust 
safety net programs accordingly. 

□ Student progress is monitored 
and assessed throughout safety net 
implementation. 

□ All instructors of safety net 
courses have appropriate content 
expertise. 

□ Teacher teams regularly meet 
to discuss progress of at-risk 
students towards standards. 

□ Students are moved in and out 
of safety net programs as needed 
to perform at standard. 

□ All classroom teachers can 
identify at-risk students in their 
classes and have a plan for 
moving the students up to 
standard. 

□ Safety net programs are 
reviewed and revised for 
effectiveness and targets are 
developed to include in following 
year’s School Improvement Plan. 


1 . Preparing 


2. Getting Started 


3. Moving Along 


4. In Place 


□ 


□ 


□ 


□ 
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Standards Implementation Snapshot System 

A system for tracking the implementation of Standards in Duval County 

Evidence Form: Safety Nets 

Questions for Principal: 

1) Please describe the safety net programs at your school? 

2) What data do you use to help you identify students 
who are at-risk and how do you analyze the data? 

3) How would you rate your school on the overall 
rubric (provide rubric)? 

4) How do you know when safety net programs are working? 

5) What training has been provided to your staff on 
helping at-risk students meet standards? 

6) Who teaches safety net courses and what are their qualifications? 

Questions for Leadership Team member: 

1) How is your school identifying and helping at-risk students? 

2) What is the role of your leadership team in the school’s safety net programs? 

3) How do you know when safety net programs are working? 

Questions for Safety Net Providers: 

1) How were you chosen to provide the safety net course? What is your content background 
in this subject? 

2) Do you talk with the students’ regular teachers? What do you talk about? How regularly 
do you talk? 

Questions for Teachers: 

1) Who are the at-risk students in your class? How do you 
know when a student needs instructional intervention? 

2) What strategies are you using to bring them up to standard? 

3) Can you show me an action plan for a struggling student? Please explain it to me. 

4) Do you meet with other teachers to discuss at-risk students? With whom? How often? 
What information do you use? What do you talk about? 

Questions for Students whom teacher has identified as at-risk: 

1) What standards are the most difficult for you to meet? 

2) What would help you to meet the standards? 

3) What kind of help are you getting to meet the standards? 

Possible school/classroom artifacts: 

Examined Present 

□ □ Safety net plans in SIP. 

□ □ Individual student monitoring forms. 

□ □ Safety net schedules. 

□ □ Safety net attendance rosters. 

□ □ Student action plans. 

□ □ Targeted student list with data. 

Other Pertinent Information: 
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Appendix B - Visit Guidelines 
and Teacher Sampling Form 

Safety Nets 
Visit Guideiines 



INFORMATION FOR YOUR VISIT 

1 . The visit will begin with a brief interview with the principal (see evidence form) and the picking of a 
leadership team member and teachers to interview. The snapshot team may choose to split up and 
conduct interviews individually (resulting in a shorter visit) or to conduct interviews and classroom 
visits together. During the training session, some groups found it helpful to interview a teacher team 
during one of their planning periods. Be opportunistic to interview teachers who have time available 
during the time of your visit. The number of individuals to talk to depends on the size of the school 
you are visiting. Use the following guidelines: 

Snapshots of small elementary schools (<700 students, 2-member snapshot team) should include 

the following: 

1 Principal interview 

1 Leadership team member interview 

3 classroom teacher or team interviews, making sure you include one ELA teacher, one 

mathematics teacher and one science teacher. The ELA and mathematics teachers or teams 
should be in grades 3 and/or 4. The science teacher should be in grade 5. 

Small High schools (<700 students, 2-member snapshot team): 

1 Principal interview 

1 Leadership team member interview 

3 classroom teacher or team interviews, making sure you include one ELA teacher, one 

mathematics teacher and one science teacher. The ELA and mathematics teachers or teams 
should be in grades 9 and/or 10. The science teacher should be in grade 10. 

Large Elementary schools (>700 students, 3-member snapshot team): 

1 Principal interview 

1 Leadership team member interview 

5 classroom teacher or team interviews, making sure you include at least one ELA teacher, one 
mathematics teacher and one science teacher. The ELA and mathematics teachers or teams 
should be in grades 4 and/or 5. The science teacher should be in grade 5. 

Large High schools (>700 students, 3-member snapshot team): 

1 Principal interview 

1 Leadership team member interview 

2 Department chairs (1 ELA, 1 Mathematics). 

3 classroom teacher or team interviews, one ELA teacher, one mathematics teacher and one 

science teacher. The ELA and mathematics teachers or teams should be in grades 9 and/or 
10. The science teacher should be in grade 10. 
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2. You may want to use the table on the following page to help you select the sample of classrooms to 
visit. 

3. Complete interviews of administrators and teachers/teacher teams, using specified questions from 
the evidence forms. 

4. Meet with your fellow snapshot data collectors and complete rubric, using evidence from your 
interviews. 

5. Debrief with principal (see guidelines on last page of this document). 

6. One member of your snapshot team should go to the website and enter the ratings from the 
observation. 
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Sampling of Teachers 



Use this form when you are picking the classrooms to be visited with the host principal. It may help you 
to array the possible classrooms to be visited and help you in selecting a representative sample of the 
school. The host principal should select one classroom and the visiting data collectors should select 3. 

• Elementary school classrooms should be in grades 3 and 4. 

• Middle school classrooms should be in Language Arts in grades 6 and 7. 

• High school classrooms should be in Language Arts in grades 9 and 1 0. 



Grade 


Teacher Name 


Comments 


Selected 
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Guidelines for Feedback to Principals 



After you have conferred with your partner(s), come to agreement on what you observed and complete 
your rating form. 

Have a brief conversation with the host principal. Start by briefly telling them what you saw, what your 
ratings were, and the evidence that you believe supports the conclusions that you have drawn. Stick with 
the facts. You may wish to go classroom by classroom, or make summary statements across classrooms. 
Allow them to question your conclusions and provide more information to help explain what you saw 
(although this should not change your ratings unless you feel that you fundamentally mis-interpreted what 
you saw). 

Remember that you are making sensitive judgments about their school, so please be as constructive as 
possible. Your conversation and your ratings are confidentiai. Piease do not discuss them with 
anybody after you have compieted the observation. 



28 





Monitoring instructional Reform 



Appendix C - Sample Snapshot Results 



Supovitz and Weathers 



100% n 
80% - 
60% - 
40% - 
20 % - 
0 % - 



Safety Nets 
Overall Ratings 



April, 2003 
All Schools {n=21) 



December, 2003 
All Schools {n=36) 



( 0 ) 



Preparing 



57% 

( 12 ) 




Getting Started Moving Along 



100% n 
80% - 
60% - 



24% 40% - 




In Place Preparing 



42% 

(15) 




Getting Started Moving Along 



31% 

( 11 ) 




In Place 
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April 2003 

Elementary (n=14) and Secondary (n=6) 




n Elementary n Secondary 



30 



Supovitz and Weathers 



100% n 
80 % - 
60 % - 
40 % - 
20 % - 
0 % - 



December 2003 

Elementary {n=26), Middle {n=6), 
& High Schools (n=4) 



4% " (5) 

(0) 


33% 

(2) 


25% 

(1) 


50% 50% 
38% (3) (2) 
(10)^ — 




38% 

(10) 


17% 

(1) 

■ (0) 


Preparing Getting Started Moving Along In Place 



D Elementary ■ Middle □ High 



